Rate control tasks in video encoding can be greatly enhanced if the rate-quantization characteristic R(QP) of the current frame is known, i.e. which quantizer yields which rate. This obviously holds, for example, for low delay rate control where it is important to match a given target bitrate very closely. However, in real-time applications it is usually not possible to determine R(QP) exactly as this would involve encoding the frame with all possible quantization parameters. Therefore, models were introduced that try to predict the relationship between rate and quantization. That is, the rate-quantization function R(QP) is modeled asR(QP)≈model(QP,β)  (1)where β is a vector containing the parameters of the model.
Sufficiently accurate models are already available for the distortion-quantization function (see [21]).
However, the task of modeling the rate-quantization relation is much more difficult.
Several different models aiming to represent the R-Q characteristic of H.264/AVC coded video frames have emerged in the literature. The most popular one (e.g. used in [11]) is the quadratic model proposed by [5]. It has two adjustable parameters and often uses MAD (mean absolute difference) to predict the new frame complexity [22]. It is defined as
                              R          ⁡                      (            QP            )                          ≈                                            β              1                        ⁢                          MAD                                                Q                  Step                                ⁡                                  (                  QP                  )                                                              +                                    β              2                        ⁢                          MAD                                                                    Q                    Step                                    ⁡                                      (                    QP                    )                                                  2                                                                        (        2        )            where the relation between quantization parameter and quantization step size (QStep) is defined in the H.264/AVC standard [14]. Some further proposals of different complexity are the linear model [12], the exponential model in [24], the ρ—domain model based on the number of non zero coefficients [8] or a piecewise defined model given by [7].
Even more essential than the potential accuracy of the R-Q model is the reliable estimation of its parameters [6]. Besides the use of statistical measures (like e.g. the previously mentioned MAD) the parameters of these models are usually determined by means of linear regression e.g. [5] [10]. Furthermore there are a few approaches using the Kalman filter e.g. [23], [4].
However all of the models mentioned previously have shortcomings concerning model accuracy, complexity, smoothness or convexity. Similarly none of the published algorithms used to determine the model parameters provides a sufficiently accurate and straightforward estimation.